Hi @kapa.ai, I'm trying to import new fields from a mongodb database. The problem is that airbyte is not updating the schema even after several rounds of "Refresh Source Schema", so I can't import those new fields. I have tried to re-create the source/connection from scratch as well as increasing the document discovery size to 20000, but no luck so far. Has there been any previous incident similar to the issue I'm facing?

Thinking completed I'm sorry, but the provided knowledge sources do not explicitly document a specific bug or incident where a successful schema discovery finishes without error but fails to detect newly added fields that exist in your sampled documents. However, community discussions and the documentation highlight several known behaviors, limitations, and similar issues with MongoDB schema discovery that explain why this might be happening: 1. Inherent Limitations of $sample Discovery When Schema Enforced is enabled, Airbyte's MongoDB connector runs a

$sample

aggregation pipeline to randomly pick documents (by default 10,000, which you increased to 20,000) to infer top-level fields [Update Capture Mode: Lookup vs Post Image; Schema discovery performance impact]. Airbyte documentation explicitly notes that because MongoDB collections are schemaless, no sample size can guarantee a complete or stable schema [Schema discovery performance impact]. If the new fields only exist in a small subset of recently added documents in a very large collection, the random

$sample

stage may completely miss them during the sampling process. You can increase the Discovery Sample Size up to 100,000 documents [Update Capture Mode: Lookup vs Post Image], but this will increase discovery time and database load. 2. Similar Issues and Past Reports • Discovery Timeouts/Silent Failures: Multiple users have reported issues where schema discovery either takes too long and times out (e.g., exiting with an error or hanging due to heavy

$sample

and aggregation pipelines), or appears to fail silently [Github Issue (closed): Mongodb connector fails to refresh schema; Forum Post: Mongo Connector Schema Discovery Fails]. • Adding Fields Manually: Other users experiencing difficulty getting the discovery process to detect new fields attempted workarounds like manually patching the catalog spec via the Airbyte API [Forum Post: Manually Add Collections to Existing Mongo Connector]. Recommended Workaround: Disable Schema Enforcement If increasing the sample size does not capture your new fields, the recommended solution is to disable Schema Enforcement (set Schema Enforced to

false

in the source configuration) [Schema discovery performance impact; Replication Methods]. When schema enforcement is disabled: • Airbyte does not run sampling to discover fields [Update Capture Mode: Lookup vs Post Image; Schema discovery performance impact]. • Every document is synced with an

_id

field and a

data

JSON blob containing the entire set of document fields [Replication Methods]:{ "_id": "document id", "data": { "entire set of fields found in document" } } • No fields are omitted regardless of when they were added or how varied the structure is across documents [Replication Methods]. You can then extract your new fields downstream in your destination. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Thanks, for others' reference, it worked after I increased the sample size to the maximum of 100,000 and also refreshed the source schema several times